Predictability and the Base Rate

Whenever a statistician wants to predict the likelihood of some event based on the available evidence, there are two main sources of information that have to be taken into account: (1) the evidence itself, for which a reliability figure has to be calculated; and (2) the likelihood of the event calculated purely in terms of relative incidence. The second figure here is the base rate. Since it is just a number, obtained by the seemingly dull process of counting, it frequently gets overlooked wh...

 How IBM's Watson Pattern-Matches to Answer Trivia

The way Watson plays the game also requires massive amounts of pattern matching. The supercomputer has been loaded with hundreds of millions of unconnected digital documents, including encyclopedias and other reference works, newspaper stories, and the Bible. When it receives a question, it immediately goes to work to figure out what is being asked (using algorithms that specialize in complex communication), then starts querying all these documents to find and match patterns in search of the ...